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a Development of a Testing System (X) 
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a A Novel Approach to Composing Test sheets for 


Multiple Assessment Criteria in Building Testing 
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= Development of A Testing System to meet Multiple 
Assessment Requirements (O) 
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In recent years, researchers have developed various computer-assisted 
testing systems to evaluate students' learning status more precisely. For 
example, Feldman and Jones attempted to perform semi-automatic testing 
of student software under Unix systems [5], Rasmussen et al. exercised a 
system to evaluate students learning status on computer networks by taking 
their progress into consideration [22]; furthermore, Chou proposed the 
CATES system [3], which is an interactive testing system developed in a 
collective and collaborative project with theoretical and practical research 


on complex technology-dependent learning environments. Unfortunately, 
although many computer-assisted testing systems have been proposed, few 


of them have addressed the problem of finding a systematic approach for 
composing test sheets satisfying multiple assessment requirements. Most 
of the existing systems construct a test sheet by manually or randomly 
selecting test items from their item banks. Such manual or random test 
item selection strategies are inefficient and are hardly able to meet multiple 
assessment requirements simultaneously. 
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Some previous investigations showed that a well-constructed test sheet 


not only helps evaluate the learning status of the students, but also 
facilitates the diagnosis of the problems embedded in the learning process 
[13, 14, 17]. That is, it is very critical to select proper test items to 
constitute a test sheet that meets multiple assessment criteria, including the 
expected time needed for answering the test sheet, the number of test items, 
the specified distribution of course concepts to be learned, and the most 
important, the maximization of the average degree of discrimination [20]. 


Since it is difficult to satisfy multiple requirements (or constraints) in 


selecting test items, most computerized testing systems generate test sheets 
in a random fashion (which will be called "random selection" 

throughout this paper) [16]. In [15], a multiple criteria test sheet generating 
problem is formulated as a dynamic programming model [11] to minimize 


the distance between the parameters (e.g., discrimination, difficulty, etc. 


of the generated test sheets and the objective values subject to the 
distribution of concept weights. 
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Although the dynamic programming approach has taken multiple 
| requirements into consideration, in practical applications, more 
criteria need to be addressed. For example, a teacher might like to 
assign a range of test times instead of giving a fixed test time and 
usually a teacher will assign an expected lower bound for each 
concept weight instead of giving a distribution of concept weights. 
Moreover, the goal of a group test is to discriminate the status of the 
students. This implies that the discrimination degree of the entire test 
sheet needs to be maximized as much as possible. Another critical 
issue arising from the use of dynamic programming approach is 
probably the exceedingly long execution time required for producing 
optimal solutions. As the time-complexity of the dynamic 
programming algorithm is exponential in terms of input data, the 
execution time will become unacceptably long if the number of 
candidate test items is large. 
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To cope with these problems, researchers attempted to formulate 

.a new test sheet-generating problem by optimizing the 
discrimination degree of the generated test sheets with a specified 
range of assessment time and some other multiple constraints [16]. 
However, the new formulation is even more complex, previous 
investigations showed that it is difficult to efficiently find near- 


optimal solutions when the number of candidate test items is larger 
then five thousand [16]. 


In the following sections, a model for formulating the problems 
of finding a set of test items that fit multiple assessment criteria is 
presented. As the problems are NP-hard, a tabu-based algorithm is 
proposed to find quality approximate solutions in an acceptable time. 
Computational experiments will be also presented to study the 
performances of the proposed algorithms. 


(DR BEER eel CRT ) 
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RA ATS FRE EO]: 


In an item bank, a subset of n candidate test items Q, 


Q, ..., Q, will be selected for composing a test sheet. With the 
eae course concepts to be learned, say C a , each test item each test item 
is relevant to one or more of them. For example, to test the 
basic computer knowledge of students, C; might be "1/0 
devices", "Central Processing Unit" or "Memory". In this section, 
a mixed integer programming model [19] is presented to 
formulate the test sheet-generating problem under multiple 
assessment considerations. The model aims at optimizing the 
discrimination degree of the generated test sheets with a 
specified range of assessment time and some other multiple 
constraints. Such a model has been called the Specified Range 
of Assessment Time (SRAT) Problem [16]. (ES IHRE HEW] ) 


In the SRAT problem, the major consideration is to confine 
the length required by the students to answer the selected 
items. The variables used in the formulated models are defined 
as follows: (公式 ) 
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Tabu search is a metaheuristic method which guides a 
local search procedure to effectively explore the solution 
space of optimization problems. |t was first introduced by 


Glover [7, 8] and has exhibited successful applications for 
solving the traveling salesman problem [2], generalized 
assigament problem [18], job shop scheduling problem [23], 
and network loading problem [1], just to name a few. (Fk 
的 来 产 ) 

“Tabu” comes from Tongan, a language of Polynesia, 
where it indicates things that cannot be touched because 
they are sacred, and the word now means prohibition [9]. 
Analogously, tabu search forbids the reverse search to revisit 
solutions already encountered. Tabu search has the following 
distinctive features which make it unique from other 


metaheuristics.... (方法 的 特性 ) 
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In the followings, a tabu search-based algorithm, MST (Bit-Map Selection 
» with Tabu approach), is proposed to find quality approximate solutions for 


the SRAT problem. 《方法 如 何 套用 到 目前 的 问题 ) 
Input: mtest items Q, Q, ..., Q, and mconcepts G, C, ..., C3 
Step 1. Generate initial configuration 


Since all decision variables of the SRAT problem take binary values 
(either O or 1), a configuration in the S7 algorithm can be represented 
in a condensed form as x = [x,x;..x ], which is a vector of n binary bits 
where xis equivalent to 1 if test item Q, is selected and O otherwise. The 
initial configuration is designated as [OO...0], i.e., all the test items are not 
selected. The reason is due to the objective of the optimization problem, 
which requests maximization of the average discrimination degree for 
selected test items. Therefore, if the algorithm starts with a zero vector 
and incrementally select test items with higher discrimination degrees by 
replacing O with 1, the search for the optimal solution could be expedited. 


Step 2. Identify the neighborhood 
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Computer skill certification test is one of the most important tests to 
evaluate the computer ability of students. As the test is held frequently, it is 


critical to efficiently and effectively compose test sheets from a large item 


bank containing over ten thousands of test items. To fairly certificate the 
computer skills of students, the composed test sheets must meet multiple 


assessment criteria, such as the ratio of relevant concepts to be evaluated, 
the average discrimination degree, difficulty degree and estimated testing 
time. It will take significantly long time to compose an optimal test sheet 
from a large item bank by generating and testing each possible candidate 
combination of test items. To cope with this problem, a tabu search-based 


approach is proposed to improve the efficiency for composing near-optimal 
test sheets from very large item banks to meet multiple assessment criteria. 
Based on the proposed approach, a computer-assisted testing system has 
been developed; moreover, a series of experiments have been conducted to 
compare the efficiency and efficacy of the novel approach with other 
approaches. From the experimental results, it can be seen that the novel 
approach is desirable in composing near-optimal test sheets from large item 


banks. (farte — EXHI UT EAN EEN) 
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In this paper, a tabu search-based approach, BMST (Bit-Map Selection 
with Tabu), is proposed to cope with the test sheet-generating problems. 
BMST algorithm has been embedded in a computer skill certification system 
with large-scale test banks that are accessible to students as well as 


instructors through the world-wide-web. ( nisu = 的 ) ) 


To evaluate the performance of the proposed algorithm, two 
experiments have been conducted to compare the execution time and the 
solution quality of four solution-seeking strategies on eighteen item banks. 
Experimental results manifest that test sheets with near-optimal 
discrimination degrees can be obtained in a much shorter time by 


employing the novel approach. ( Deas fo EJER ) 


For further extensive applications, some collaborative plans with some 
local electronic companies are proceeding, in which the novel approach is 
applied to test and train junior engineers, such as the knowledge and skills 
of Nano Self-Assembly, Quantum Dot Self-Assembly, Copolymer Self- 
Assembly, Electrostatic Self-Assembly, etc. 
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= Based on the experimental results, it can be seen 
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fetus s-e-mail 


Dear Prof. Gonzalez, 


Attached please find our manuscript entitled "A Concept-based 
Cooperative Learning Approach for Science Courses" submitted to Second 
International Conference on Multimedia and ICTs in Education (m-ICTE 
2003) for possible presentation. Your acknowledge will be highly appreciated. 


Thank you. 


Sincerely, 
Gwo-Jen Hwang 


Professor of Information Management Department 
National Chi Nan University 
Pu-Li, Nan-Tou, Taiwan 545, R. O. C. 
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Dear Prof. Gwo-Jen Hwang, 


The Editor of the Transactions on Education acknowledges receipt of the 
following manuscript: 


No. TE-2003-000262-A Concept-based Approach to Conducting 
Cooperative Learning Process 


It is understood that this manuscript is entirely original, has not been 
copyrighted, published, submitted, or accepted for publication elsewhere, 
and all necessary clearances and releases have been obtained. If the 
material in this paper has been published before in any form, it is imperative 
that you inform me immediately. 


You will be notified via email when the review of this manuscript is 
completed. Please refer to the paper number in any communications 
regarding your manuscript. You may check the review status of your 
manuscript via the IEEE manuscript Central website. When the review of 
your manuscript has been completed, you will be notified of its disposition by 
email and at that time reviewer comments will also be made available to you. 


Sincerely, 
Editor-in-Chief 
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(EIC reject - format revisions) 


Prof. Hwang: 


| regret to inform you that | have made the decision to reject your 
manuscript, TE-2004-000296, titled "A Tabu Search Approach to Generating 
Test Sheets for Multiple Assessment Criteria" IN ITS PRESENT FORM so that 
you can make adjustments in the manuscript and submit a revised version 
for formal review. 


The Transactions now requires that all manuscript content be in the 
English language. Your manuscript, as submitted, contains graphical 
information that is in a language other than English. Please replace those 
graphics containing non-English content with graphics totally in English. 


Once you have made these substitutions, please submit your revised 
manuscript for review. 


Thank you in advance for your attention to this matter. 
Sincerely, 
David A. Conner, Ph.D., P.E. 


Editor-in-Chief 
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Dear Dr. S. Heller, 


Attached please the revised manuscript " A Group-Decision 
Approach for Evaluating Educational Web Sites" submitted to 
computers & Education for possible publication. A file containing the 
revision Summary is also attached. Your acknowledgement will be 
highly appreciated. 


Thank you. 


Sincerely yours 

Gwo-Jen Hwang 

Information Management Department 
National Chi Nan University 

Pu-Li, Nan-Tou, Taiwan 545, R.O.C. 
FAX: 886-940503178 

TEL: 886-915396558 
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(ECAC Sit A -Revision Summary 


Response to Reviewers and Editor 
Paper#: SMCC-03-06-0056 


Title: On the Development of a Computer-Assisted Testing System 
with Genetic Test Sheet-Generating Approach 


[Reviewer 1 Comments]: 
The paper should be shortened. 
[Response to Reviewer 1]: 


The paper has been shortened to 24 pages by removing some 
redundant descriptions of genetic models and algorithms; moreover, 
Sections 3 and 4 have been re-written to condense the entire paper. 
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修改 文件 ft HA Revision Summary 


[Reviewer 2 Comments]: 


No innovative contribution was found both in the theory of genetic 
algorithms and in the application of them. 


[Response to Reviewer 2]: 


(1) We have re-written the abstract and Sections 1 and 2 to explain 
the importance about the construction of a good test sheet. The 
major contribution of this paper is not in its technical part. Instead, we 
tried to cope with an important problem arising from real educational 
applications. Such a problem is known to be critical and has not been 
efficiently and effectively solved before. 


(2) Since the innovative contribution of this paper might not be 
significant, we have re-written the paper as a technical 
correspondence based on the editor's suggestion. 
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修改 文件 ft HA Revision Summary 


[Reviewer 3 Comments]: 


Make the definitions, formulas, and other descriptions clearer and 
more precise, so that the revised paper will be improved in its 
readability and correctness. 


[Response to Reviewer 3]: 


Te mixed integer models and the genetic algorithms in Sections 3 
and 4 have been re-written to make the definitions, formulas, and 
other descriptions clearer and more precise (please refer to Pages 6- 
17). Moreover, a colleague who is an English expert has carefully 
checked the paper to correct potential grammatical errors. 
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Dear Prof. Hwang, 


The review of your manuscript titled "A Test Sheet- Generating 
Algorithm for Multiple Assessment Requirements", TE-2001-000029.R1, 
has been completed. The Editorial Review Board, while feeling that 

our manuscript is worthy of publication, feels that the manuscript 
requires a few minor revision before it can be published in the 
Transactions on Education. 


When reviewing the manuscript, each reviewer was asked to complete 
two questionnaires: one that directed comments to the Editorial Board 
and one that directed comments to the author(s) of the 

manuscript. You will find attached reviewer comments directed to 
you. In preparing the next revision of your manuscript, you will be 
expected to address the issues brought up by the reviewers. Once you 
have addressed these issues, your manuscript will be ready to 


processed to publication. 
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_ In preparing your next revision, here are some important criteria that need 
to be followed. 
l. The next revision of your manuscript should be double-spaced and 
typed in 12-point type. DO NOT single-space! DO NOT use a two- 
column, journal-style format! 
2. Make sure that all bibliographical references for books and published 
papers follow the I EEE format described at 
http: // standards. ieee. org/ guides/style/section7.html#992. 
3. Make sure that all bibliographical references to specific information on a 
web site include the last date the site was viewed. Web sites change and 
specific content is often removed. A reader requiring specific, referenced 
content that 
has been removed will need a date on which the information appeared 
when requesting that information from the Web Master's archive. 
4. Number each page. 
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When you have completed revising your manuscript, mail a copy to the 
Transactions' Editorial Administrator, J erry Ann Conner. Be sure to include 
your manuscript number in your cover letter when you transmit your 
manuscript to Mrs. Conner. When Mrs. Conner receives your revised 
manuscript, a review will be made to ensure that appropriate revisions 
have been made. Then the manuscript will be reviewed for English 
grammar, punctuation, spelling, and correct word usage. When these 


reviews have been completed, the Transactions' Editorial Administrator 
will mail you two items: (1) a copy of your manuscript that has been 
edited for grammar, punctuation, spelling, and correct word usage and (2) 
a check list of required items that must be submitted for your manuscript 
to be published. Your final manuscript should include all corrections 
noted by the Editorial Administrator. When all items are ready for 
submission, they should be mailed to the address indicated on the check 


THREE 


Thank you for your interest in publishing in the Transactions. 
We look forward to seeing your manuscript in print. 


Sincerely, 
David A. Conner, Ph.D., P.E. 
Editor-in-Chief 
IEEE Transactions on Education 
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Behr 


Dear Dr. Hwang, 


The proof for your article, A Test-Sheet- Generating Algorithm for 
Multiple Assessment Requirements, is ready for your review. Please 
connect to the following URL to retrieve your proof: 

URL: 
https:// authorproof.ieee.org/tfa/ authproof.do?journal-TE&artl D=46te03- 
hwang 

If you have trouble accessing your proof site, please make sure that 
your browser's security setting utilizes 128-bit SSL as it is necessary for 
access to your proof. 

If you have any problems or questions regarding this proof, please 
contact your I EEE staff editor by replying to this message. 


Christine Vartanian 

Associate Editor, 

Transactions on Education/ 
Journal of Lightwave Technology, 
I EEE Publishing 
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Behr 


ear Dr. Hwang, 

We have received your above-mentioned article for publication. On 
behalf of Elsevier Science, I would like to take this opportunity to thank you 
for choosing Computers & Education as your publishing medium. 

From the details supplied by the journal editor we have logged your 
address, and your e-mail, phone and fax numbers if available. Please check 
that the details are correct so we can contact you quickly, if necessary. 

Any attachment to this e-mail is in PDF format. To view and print an 
attachment you will need Acrobat Reader from Adobe. This program is 
freely available and can be downloaded from http: //www.adobe.cony. The 
Acrobat reader is available for whole series of platforms which include PC, 
Mac and Unix. If you would prefer to receive the forms by fax or mail then 
please inform us immediately by replying to this e-mail with full fax details. 

If any questions or problems arise, please do not hesitate to contact us, 
preferably by fax or e-mail quoting CAE 651 in all correspondence. 


Yours sincerely, 


Elsevier 


J. Beskeen "RET 
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Dear Prof. White, 


Recently, we checked the status of our paper " On the Development of 
a Computer-Assisted Testing System with Genetic Test Sheet-Generating 
Approach" (SMCC-03-06-0056.R3) submitted to SMCC on March 29, 2004, 
and found that the review record remained "Awaiting Reviewer Assignment". 
We are sending this e-mail to inquiry the status of the paper. Your 
acknowledgement will be highly appreciated. 


Thank you. 


Best regards, 

Gwo-J en Hwang, Ph.D. 

Professor of I nformation Management Department 
National Chi Nan University 

Web: www.im.ncnu.edu.tw/ —gj hwang 

TEL: 886-915396558 

FAX: 886-49-2915205 
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To Whom It May Concenr, 


We received an acceptance letter, dated on November 7, 2003, 
of our article submitted to Computers & Education. As we didn't 
receive further information on the preparation of our manuscript for 
publication, we sent a mail to the Edior-in- Chief to see whether our 
article has an ID for inquiry. She said no ID is needed. Do you have 
any information on the publication progress of our paper (Author: 
Hwang, Tseng and Lin; Title: An effective approach for test-sheet 
composition for large scale item banks)? Or, please advise the 
process we need to follow. 


Best regards, 
Bertrand M.T. Lin 
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= PRAKA Reference] E EP DE SE 
= 询 文 中 的 图 REEE 最 好 在 进 
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